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Abstract 

This paper first presents necessary and sufficient conditions for the solvability of discrete time, mean- 
field, stochastic linear-quadratic optimal control problems. Then, by introducing several sequences of 
bounded linear operators, the problem becomes an operator stochastic LQ problem, in which the optimal 
control is a linear state feedback. Furthermore, from the form of the optimal control, the problem changes 
to a matrix dynamic optimization problem. Solving this optimization problem, we obtain the optimal 
feedback gain and thus the optimal control. Finally, by completing the square, the optimality of the 
above control is validated. 



1 Introduction 

In this paper we consider a class of stochastic linear-quadratic (LQ) optimal control problems of mean-field 
type. The system equation is the following linear stochastic difference equation with k G {0, 1, 2, A^— 1} = 

N, 

Xk+i = [AkXk + AkExk + BkUk + BkEuk) + {CkXk + Ck^Xk + DkUk + DkEuk)wk, ,^ 

where Ak, Ak,Ck,Ck € R"^", and Bk, Bk, D, € M"^^'" are given deterministic matrices, and E is the 
expectation operator. Denote {0,1,2,..., N} by N. In (1.1) , {xk,k G N} and {uk,k £ N} are the state 
process and control process, respectively. {wk,k e N}, defined on a probability space {n,T,P), represents 
the stochastic disturbances, which is assumed to be a martingale difference sequence 

EK+il-^fc] = 0, Eliwk+iflJ'k] = 1, (1.2) 

where Tk is the c-algebra generated by {C,wi,l = 0, 1, • • • ,fc}. The initial value C and {wk,k e N} are 
assumed to be independent of each other. The cost functional associated with (1.1) is 



J(C,u)= E 



'N-l 



J2 {^iQkXk + {ExkfQk^Xk + ulRkUk + {EukfRk^Uk) 



k=0 



(1.3) 



+E (x^Gnxn) + {ExnVGnExn, 
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where Qfe, Qfc, Rk, Rk, fc G N, Gat, Gn are deterministic symmetric matrices with appropriate dimensions. 
We introduce the foUowing admissible control set 

Uad^ {u= iua,ui,--- ,UN-i) : N x ^ M™ \ Uk £ J'k,^Uk\'^ < 00} . 
The optimal control problem considered in this paper is stated as follows: 
Problem (MF-LQ). For any given square-integrable initial value C, find u° G Uad such that 

J{i:,u°)= inf J(C,u). (1.4) 

ueUad 

We then call u° an optimal control for Problem (MF-LQ). 

Unlike the classical stochastic LQ problem, the expectation Ex^ of Xk appears in the system equation 
(1.1) and the cost functional (1.3). Therefore, the above problem is called a mean-field LQ problem. This 
problem is a combination of mean field theory and a LQ problem. Mean-field theory was developed to study 
the collective behaviors resulting from individuals' mutual interactions in various physical and sociological 
dynamic systems. According to mean-field theory, the interactions among agents arc modelled by a mean- 
field term. Letting the number of individuals go to the infinity, the mean-field term will approach the 
expectation. To see this, assume the dynamics of particle i (i = 1, • ■ • , L) are 

^ L ^ L 

^fc+i = (^fe-^fc'^ ^ ^^Y^'^ BkUk^ + {Ckx'',:^ + ^'^^^ X] ^fe^ + DkUk^Wk. 

Under appropriate conditions and letting L — >■ 00, we have 

Xk+i = (AkXk + AkExk + BkUk) + [C'kXk + CkE.Xk + DkUk)wk. (1.5) 

An exact derivation of (1.5) follows the classical Mckean- Vlasov argument; for the Mckean- Vlasov argument, 
readers may see, for example, [7][28][30] and references therein. Clearly, (1.1) is a natural extension of (1.5). 
For the motivation for including Ex^ and Eufc in the cost functional (1.3), [32] points out that it is natural 
to introduce variations var(a;fc) and var(ufc) to the cost functional so as to the state process and the control 
process could be not too sensitive to the random events. An example of this case is the known finance 
mean-variance problem. For system (1.1) without mean-field terms, we refer to papers such as [3] [22] [23] [26]. 

The continuous-time counterpart of (1.1) is a mean-field stochastic differential equation (SDE), whose 
investigation goes back to the McKean- Vlasov SDE proposed in 1950s ([27] [28]). Since then, many researches 
study McKean- Vlasov type SDEs and applications ([12] [14] [15] [16]). For recent development of mean-field 
SDEs, readers may refer to [10] [11] [13] and references therein. The control problems for mean-field SDEs are 
investigated by many authors; see, for example, [1] [2] [4] [7] [9] [29] [32] [17]. On the other hand, another class 
of problems is mean-field games in terms of its ability to model the collective behavior of individuals due to 
their mutual interactions. Comparing the above mentioned papers, this class of problems may be viewed as 
decentralized control problems, that is, the controls are selected to achieve each individual's own goal. For 
other aspects of mean field games, readers may refer to [7] [18] [19] [20] [21] [24] [25] [31] and references therein. 

The problem considered in this paper is related to that of [32], which deals with a continuous time 
mean-field LQ problem. In [32], by a variational method, the optimality system is derived, which is a mean- 
field forward-backward SDE. Furthermore, by a decoupling technique, two Riccati differential equations are 
obtained. This gives the feedback representation of the optimal control. Our discussion of the mean-field 
LQ optimal control problem differs from [32] in the following ways. First, we considers the discrete time 
case. There are several situations when it is necessary, or natural, to describe a system by a discrete time 
model. A typical case is when the signal values are only available for measurement or manipulation at certain 
times, for example, because a continuous time system is sampled at certain times. Another case arises from 
discretization of the dynamics of continuous time problems. The second difference between this paper and 
[32] is that methodology differs. In this paper, we first convert Problem (MF-LQ) to a quadratic optimization 
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problem in Hilbcrt space. This gives necessary and sufficient conditions for the solvability of Problem (MF- 
LQ). The optimal control obtained in this case has an abstract form, which may be viewed as open loop. 
Secondly, by introducing several sequences of bounded linear operators, Problem (MF-LQ) becomes an 
operator LQ optimal control problem. This operator LQ problem shows that the optimal control is a linear 
state feedback, which is clearly closed-loop. Thirdly, by the linearity of the optimal control. Problem (MF- 
LQ) becomes a matrix dynamic optimization problem. Using the matrix minimum principle ([5]), we derive 
the optimal feedback gains and thus the optimal control is obtained. Finally, by completing the square, we 
validate the correctness of the obtained optimal control. 

The paper is organized as follows. The next section presents the preliminaries and two abstract considerations — 
quadratic optimization in Hilbert space and the operator LQ problem. In Section 3, optimal control via 
Riccati equations is presented. Section 4 gives an example to calculate the solutions to Riccati equations 
and the optimal feedback gains. Section 5 gives some conclude remarks. 



2 Preliminaries and abstract considerations 



Some standard notion is introduced. 



Definition 2.1 (i). Problem (MF-LQ) is said to be finite for ( if 

inf J(C, u) > —CO. 

UGUad 

Problem (MF-LQ) is said to be finite if it is finite for any ^. 

(a). Problem (MF-LQ) is said to be (uniquely) solvable for if there exists a (unique) u° G Had such 
that (1-4) holds for C. Problem (MF-LQ) is said to be (uniquely) solvable if it is solvable for any 



In this subsection, we shall consider Problem (MF-LQ) using two methods. The first converts Problem 
(MF-LQ) to a quadratic optimization problem in Hilbert space, which gives necessary and sufficient condi- 
tions for the solvability of Problem (MF-LQ). The second considers Problem (MF-LQ) in the language of 
an operator LQ problem. This reveals that the optimal control is a linear state feedback. 

Introduce the following spaces: 

Xk = L^jr^ (M") ^ {^-.n^ is J-fe-mcasurablc, Ej^p < oo} , fc e N, 

X[0,k] — ^{xo, ■ ■ ■ ,Xk) Xk G Xk, and is J"fe-measurable, X]f=o^N'P < ^'^l ' keN, 
Uk = L'^jr^iMT) = {-q : n ^ Mr\-q IS J"fc-measurable, EjTyp < oo} , fc e N. 
Clearly, for any G N, ^ G N, X[Q, k] and H = XkMi are Hilbert spaces under the usual inner products 

(a;, ^) = E {;x^ z) , for any x,z eTi, (2.1) 

and 

(x, z) = E '^XpZp , for any x = {xq, ...,Xk), z = {zq, ...,Zk) G ^"[0, k]. (2.2) 

For any variable z in Xk or Uk, the expectation of z, i.e., Ez, is clearly well defined. If we consider E as an 
operator, it is clear that the domain and range of E may differ from place to place. For example, the domain 
may be X)., Uk, and the range is R" and R™, respectively. Therefore, the domain and range of the adjoint 
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operator E* of E may differ. For example, if E maps Xk to M", then E* is defined from M" to Xk and is 
defined as 



If E maps Uk to 



(E*r„ , x) = (r„ , Ex) , for any eW,x e Xk- 
then E* is defined from W" to Uk , and is defined as 

(E* r„ , = (r„, , Ez) , for any r„ &R"\zeUk. 



Consequently, E and E* could be denoted by E-^ and E^ in order to emphasize H, which is the domain of 
E and thus the range of E*. In this paper, Ji may be Xk,Uk,k £ N, and E^ always appears accompanying 
E-H in the form of K'^KE-^ with K being a generic square matrix. Clearly, for any z £ H, E^KE-^z is in 
H. Another form is E^^ME;^^, fc e N, with M e M™^". To simplify the notation, throughout this paper, 
E^KE-HZ with z en, and E^^ME^-.x with x G Xk, will be denoted by E*KEz and E*MEx, respectively. 
The meanings will be understood during the context. 



Let 



Taking expectations in (I.l), we have 

r Exk+i = (^fc + AkjExk + {Bk + Bk)Euk, fc e N, 
\ Exo = EC. 

r ^k,l) = {Ak + Ak){Ak-i + Ak^i)---iAi + Ai), k>l, 
\ ^k,l) = /, k<l. 

Then we have 

k 

Exk+1 = ^{k, 0)EC + J2 0(Si-i + Bi-i)Eui-i, keN\ {0}. 
(=1 

On the other hand, let 

r <i>{k,l) ^ {Ak+WkCk){Ak-i+Wk-iCk-i)---{Ai^i+wi^iCi-i), k>l, 

\ $(fc,o ^i,k <i. 

By (I.l), we have for A: G N 

fe 



(2.3) 



Xk+i = <i>{k, 0)C + *(^' [i^i-i + wi-iCi-i)Exi^i + {Bi^i + wi-iDi^i)ui^i + (S,_i + wi-iDi^i)Eui^i] 
1=1 

k 

= $(fc, 0)C + J2 *(^' + wi-iCi-iMl - 2, 0)EC 



1=1 



1=1 

k 



l~2 



$(fc, 0(Az_i + wi^iCi^i) Hi - 2, ^){B,^l + B,^i)E^ 



- 0(Si-i + wi.iDi^i)ui^i + Y Hk, l){Bi-i + A-i)Eu,_i. 



1=1 



1=1 
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Now define the following operators for any C £ Aq, " G l^ad'- 

' (rc)(-) = $(•-!, o)C, 
fc = (re)(iv), 

(rc)(-) = E/il - 1, 0(^/-i + ^i-yCi.^w ~ 2, o)EC, 
fc = (fc)(A^), 

Lu = {Lu){N), 
, Lii = {Lu){N). 

Then 

= (rc)(fc) + (f c)(fc) + + 

Clearly, the operators 

T,t:?^o^X[0,N], T-.Xo^Xn, ,2 4) 

are all bounded and linear. Notiee that the spaees in (2.4) are all Hilbert space. Therefore, the corresponding 
adjoint operators uniquely exist. Further, in what follows, we use the convention 

{Qx)i-) ^QkX., VxeX[0,N-l], 

iQv)i-) = Qw-, Vv3 = ((po, • • ■ , ^N~i) with ipk e K" such that Y.k=o IV'fcP < 

{Ru){-) = RkU., \fueUad, 

iR^)i-) = Rk^; VV- = (V-o, • • ■ ,^N-i) with Vfc e R™ such that J2k=o iV'fcP < oo. 
Consequently, the cost functional J(C, u) has the following form 

J(C, It) = (Q(rc + f c + Lu + Lu), rc + f c + + Lu) + {Ru, u) 

+ {gE(rC + f C + Lu + Lu),E{TC + fC + Lu + Lu)) + {REu, Eu) 
+ (GT(f C + f C + Lu + Lu), f C 

+f C + Lu + Lu) + (GTE(fC + rC + Lu + Lu) , E(f C + f C + + Lu)) 

= {eiu,u) + 2{e2^,u) + {03^,0- 

Here 

Oi = i? + E*i?E + (L + Z)*Q(X + L) + (i + L)*E*QE(L + L) + (L + L)*G{L + L) 
+ (L + L)*E*GE(i + l), 

62 = (i + -L)*Q(r + f ) + (L + L)*E*QE(r + f ) + (L + L)*GT{f + f ) + (L + l)*E*G'TE(f + f ), 

63 = (r + f )*Q(r + f ) + (r + r)*E*gE(r + f ) + (f + t)*GT{t + f ) + (f + f )*E*GTE(f + f ), 

and the inner products are understood from the context. Hence, for any m i— > J{C,, u) is a quadratic 
functional on the Hilbert space Uad, and the original problem (MF-LQ) is transformed to a minimization 
problem of a functional over Uad- We then have the following results. 
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Proposition 2.1 (i). IJ J(C,,u) has a minimum, then 

Gi > 0. 



(ii). Problem (MF-LQ) is (uniquely) solvable if and only if Qi > and there exists a (unique) u such 



that 



Giu + eaC^O. 



(Hi). If Qi > 0, then for any C,, J{C,, u) admits a pathwise unique minimizer u° given by 

<. = -(er'e2C)(fc), ken. (2.5) 

In addition, if 

Qk,Qk + Qk>0, Rk,Rk + Rk>0, keN, Gn,Gn + Gn>0, (2.6) 

then 81 > 0. 

Proof. The proofs oi (i), (ii) and the first part of (Hi) are weU known, and are omitted here; readers may 
refer to [32] [33] [34] for solutions of similar problems of quadratic functional optimization on Hilbert space. 
Clearly, from (2.6), we have that (L + L)*Q{L + L) + {L + L)*E*QE{L + L) + (L + L)*G{L + L) > 0. Also 
for any nonzero u G Uad, 



{Ru,u) + (E*i?Eu,w) = ^ K [ulRkUk + {Euk)'^ Rk^Uk] 



k=0 

\T r, t TV? \ . nn< \T , 



= ^ [e [(life - Ewfe)' Rk {uk - Eufe) + (Eufe)' {Rk + Rk) (Eufe) 

k=0 



> 0. 



This implies that R + WRE is positive definite. Therefore, Qi is positive definite. This completes the proof. 
□ 



Remark 2.1 Proposition 2.1 presents necessary and sufficient conditions for the solvability of Problem (MF- 
LQ); it also shows that the optimal control (2.5) is a linear functional of initial value C- Note that for any 
k G N, Uk does not depend on the current state Xk explicitly, and it depends on C and k. Therefore, the 
optimal control (2.5) may be viewed as an open-loop control. 



We shall show that under condition (2.6), the unique optimal control (2.5) is, indeed, a linear feedback 
of current state, i.e., a closed-loop control. First, introduce several sequences of operators 

AkX = AkX + AfeEx, x e Xk A: e N, 
BkU = BkU + BfeEu, u eUk, k eN, 

(2.7) 

Ckx = Ckx + GkEx, X e Xk, fc e N, 
Vku^ Dku + DkEu, ueUk, keN. 

Clearly, Ak,Bk,Ck,'Dk:k G N, are all bounded linear operators, defined from Xk and Uk to Xk and Uk, 
respectively. Consequently, (1.1) may be rewritten as 

Xk+i = [AkXk + BkUk) + {CkXk + VkUk)wk. (2.8) 
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Further, the performance functional may be represented as 



kXk,Xk) + (QfeExfejEaTfe) + {RkUk,Uk) + {Rk^Uk,Euk)) 



k=0 



+ (GnXn,Xn) + (Gat Ex AT, Ex Af) 



J2 (((Qfc +E*OfcEK,.Tfc) + {{Rk+E*RkE)uk,Uk)) + {{Gn +E*GnE)xn,xn). 



fe=0 



Here {QkXk, Xk) denotes E (xJ^QkXk) , (Qfc+E*QfcE)xfc = QkXk+E*QkE,Xk, with similar meanings for related 



notation. Define 



QkX = {Qk+E*QkE)x, xeXk, keN, 
UkU = {Rk + E*RkE)u, u £ Uk, keN, 
Gnx = {Gn + E*GArE)x, X e Xn- 



Then 

N-l 



J(C,u) = ^ {{QkXk,Xk) + {UkUk.Uk)) + {QnXn,xn)- (2.9) 



fe=0 

Therefore, Problem (MF-LQ) may be rewritten as the following 

minimize (2.9) 



subject to u G Uad, with (x.,u.) satisfying (2.8). (2-10) 

Clearly, (2.10) is an operator stochastic LQ problem in discrete time. Operator LQ problems in the de- 
terministic and continuous-time cases have been studied thoroughly. Generally speaking, an operator LQ 
problem is a problem of infinite dimensional control theory. For general infinite dimensional control theory, 
readers may refer to [6] and related literature. In this paper, by transforming Problem (MF-LQ) to (2.10), 
it is possible to obtain the closed-loop form of the optimal control. 

Suppose we have a sequence of self- adjoint bounded linear operators {Vk '■ Xk ^-^ Xk', k £ N}, which are 
determined below. Noting that Xi C Aj,, for / < fc, we assume that 

VkXi = {Vkx \xeXi]£Xu l<k. (2.11) 

This will be proved below. By adding {Vnxn,xn) — {VoXojXq) to both sides of (2.9), we have 

J(C,m) + {Pnxn,xn) - {PqXo,xo) 



= T.k=0 {{QkXk,Xk) + {TZkUk,Uk) + {Pk+lXk+l,Xk+l) - {'PkXk,Xk)) + {gNXN,XN). 

To proceed, we need some calculations. 

k+lXk+l, Xk+l) 

= (Vk+i[iAkXk + BkUk) + [CkXk + 'DkUk)wk], [{AkXk + BkUk) + {CkXk + 'DkUk)wk\) 

= E [{AkXk + BkUkY {Vk+MkXk + BkUk))] + E [{AkXk + BkUk)^ {Vk+i{CkXk + VkUk)wk)] 

+E [{CkXk+VkUk)'^ {Vk+i{AkXk + BkUk)wk)\ + E[{CkXk+VkUk)'^ {Vk+i{CkXk + VkUk)wl)] 



(2.12) 
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Clearly, by (2.11), we have 

E [{CkXk + VkUkY {Vk+i{AkXk + BkUk)wk)\ 

= E [E {wk\Fk-i) {CkXk + VkUkf iVk+i{AkXk + BkUk))] = 0, 
E [{AkXk + BkUkY (Vk+i{CkXk + VkUk)) Wk] 

= E [E {wk\Fk-i) {AkXk + BkUkY (Vk+iiCkXk + VkUk))] = 0, 
E [{CkXk + VkUkY {Vk+iiCkXk + VkUk)wl)] 

= E [E {wl\Tk~i) {CkXk + VkUkY {Vk+i{CkXk + VkUk))] 
= E [{CkXk + 'DkUk)'^'Pk+i{CkXk + VkUk)] . 
Therefore, it follows that 

{Pk+iXk+i,Xk+i) = E [{AkXk + BkUk)^Vk+i{AkXk + BkUk) + {CkXk + VkUk)'^Vk+i{CkXk + VkUk)] 

= {AlVk+iAkXk,Xk) + {Cl'Pk+iCkXk,Xk) + {AlVk+iBkUk,Xk) + {B'l'Pk+iAkXk,Uk) (2.13) 
+ {Cl'Pk+iVkUk,Xk) + {VlVk+iCkXk,Uk) + {BlVk+iBkUk,Uk) + {Vl'Pk+i'DkUk.Uk) ■ 
Substituting (2.13) in (2.12), we have 

J{xo,u) = [{{Qk + A*kVk+iAk+ClPk+iCk-Pk)xk,Xk) + 2{{BlVk+iAk + VlVk+iCk)xk,Uk) 

+ {{TZk + BlVk+iBk + VlVk+iVk)uk,Uk)] + {{Qn - Vn) xn, xn) + {Pqxo, xq) (2.14) 
= ^ [{OokXk,Xk) + 2{QikXk,Uk) + {&2kUk,Uk)] + {{Qn - 'Pn)xn,xn) + {VoXo,xo), 



k=0 

where for any k 

Qok = Qk + AlVk+iAk + ClVk+iCk - Vk, 

Qik = BlVk+iAk + VlVk+iCk, (2.15) 
e2k =Tlk + BlVk+iBk + VlVk+iVk. 

Consequently, we have the following result: 

Proposition 2.2 Under the condition that 

Qk,Qk + Qk>Q, Rk,Rk + Rk>Q, k eN, Gn,Gn + Gn>Q, (2.16) 
the unique optimal control for Problem (MF-LQ) is 

ul = - {TZk + BlVk+iBk + V*kVk+iPk)'' {BlVk+iAk + VlVk+iCk) Xk, (2.17) 

where 

( Vk= Qk + AlVk+iAk + ClVk+iCk 

- {BlVk+iAk + VlVk+iCkT {Uk + BlVk+iBk + VlVk+M'^ {B^Vk+iAk + VlVk+iCk) , k e N,(2.18) 
Vn = Gn- 



Proof. By (2.14), if (2.18) is well posed, we have that 



Af-l 



J(C, m) = ^ [{Q2k{uk + e2fe GifcXfc), {uk + Q^^QikXk)) + (Oofc - ej^e2fc GifcXfe, Xk)] 



k=0 



+ {{Gn - Vn) xn,xn) + {VoC, C) 



AT-l 



[{e2k{uk + e2k^ikXk), {uk + e-^BikXk))] + {VoCO- 



k=0 
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Therefore, the optimal control is given by ul = -Q^^^QikXk, e N, which is (2.17). We shall now prove 
that (2.18) is well posed. First, for any fc G N and Uk G Uk and Uk 7^ 0, it is clear that 

{UkUk^Uk) = E {ulRkUk) + (Eufc)^i?fc(Eufc) 

= E \{uk - Eiife)^ Rk {uk - Ewfc)] + {Kukf {Rk + Rk) {^Uk) 

> x[''^E\uk - Eukll, + Xif' \Euk\l, 
= X['^ {E\uk\l - \Euk\l) + Xi'^ \Euk\l 

> AW(EK|2„-|E«fc|2„ + |Eufc|2J 



(2.19) 



Here | • \m denotes the norm in K™; x['^\x''2^ are the smallest eigenvalues of matrix Rk and Rk + Rk, 
respectively, and A^'^^ = minjA^'^^ Aj''^}; || • \\m is the norm induced by inner product in Uk- By (2.19), we 
know that TZk > A^*"''/, where / is the identical operator defined on Uk, for each k £ N. Under assumption 
(2.16), we know that A^*"'^ > 0. Thus we have that TZk is positive definite. Furthermore, by known results, 
we have that 

7^fc = {Rk+E*RkEy = R*k + (E*i?feE)* = Rl+E*Rl (E*)* = Rk+E*RkE = Uk, 

which means that TZk is self-adjoint. Similarly, we have that Qj^ is self-adjoint and positive semi-definite. 
Therefore, it follows 

{(TZn-1 + BI^_{PnBn-i + 'D*f^^iVN'DN-i) un-i,un-i) 
= {nN-iUN-i,UN-i) + {Gn (Bn-iun-i) , Bn-iun-i) + {Qn {Vn-iUn-i) , Vn-iUn-i) (2.20) 

> {nN-lUN-l,UN-l)>X^^-^mUk\\l,. 

Then, Q2{n-i) = T^n~i+B'^_i'PnBn^i+V^_-^'PnT^n-i'^s self-adjoint and positive definite, and invertible. 
Clearly, 62(Ar-i) is bounded and linear. By the inverse operator theorem, we have that (^2{n-i) '^^ bounded 
and linear. Consequently, (2.18) is well-posed for fc = — 1. By conditions in the paragraph before (2.3), 
we known that for any Z < - 1, z G A",, K G M"""" and AI G M™^", we have that E*A'Ez G Xi and 
E*MEz G Uk- Therefore, 



{B%_,VNAN~i+'D%_,VNCN-i)zeUi, zeXi, 1<N-1, 



and 



{nN-i+B*N_iVNBN-i+V*i^_^PNT^N-i)zeUi, zeUi, 1<N-1. 
From (2.22), it follows that 

{nN^i+B%_^VNBN-i + V%_^PNVN-iy^ z eUi, zeUi, l<N~l. 
By (2.21) and (2.23), we have that u'^_-^ is J^Ar_i-adapted and thus in Un-i- Further, we have 

Vn-iz G Xi, for any z G A"/, ^ < - 1, 
which is (2.11) for fc = — 1. Similar to (2.14), we have for any a;Ar_i G J-'n-i 



(2.21) 
(2.22) 

(2.23) 
(2.24) 



xJj_iQn~ixn-i + (ExN-i) Qn~iExn-i + uJj_j^Rn-iun-i + (Emat-i) Rn-iEun-i 



-E 



_j^Rn-iun-i + (EuN-i) Rn-iEun~i 



> {Vn-ixn-i,xn-i), 
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and the equality is achieved by u'^_^. This implies that 

{Vn-1XN-1,Xn-i) 



E 



> 0. 



Thus, Vn-1 is positive semi-definite. Clearly, by (2.18), Vn-i is linear and self-adjoint. Now, we are able to 
prove that Vn-i is bounded. As noted by above, ^2(n-i) bounded, so we can easily assert that Vn-i is 
bounded. Therefore, we may prove by induction that {Vk: A: S N}, are all self-adjoint, positive semi-definite 
and bounded linear operators, and that the optimal control is given by (2.17) with G Uk^ k G N. This 
completes the proof. □ 



3 Solution by Riccati equations 

The results presented in previous section are mathematically pleasing, but they are not in a form which 
can be implemented, as (2.5) and the operator Riccati difference equation (2.18) are referenced. However, 
Proposition 2.2 provides us with the information that the optimal control of Problem (MF-LQ) is a linear 
state feedback of operator form. Therefore, it is reasonable to assume that the optimal control u° takes the 
form 

< = Llxk + LlExk, k e N, (3.1) 

with L1,L'^ G R™^". To compute the optimal feedback gains L1,L1, we start from a generic linear feedback 
control 

Uk=LkXk + LkExk, Lfe,ZfcGM"'^", ken. (3.2) 
Under (3.2), the closed loop system (1.1) becomes 

Xk+i = [{Ak + BkLk)xk + [Bklk + Ak + Bk{Lk + Lk)]Exk] 

+ [{Ck + DkLk)xk + [DkLk + Ck + Dk{Lk + Lk)]Exk] Wk, (3.3) 
xo = C, 
and the cost functional (1.3) is 

N-l 

J(C, u)= ^ E [xlQkXk + (ExkfQk^Xk + (LkXk + LkExkf Rk{LkXk + LkExk) 



k=0 



~{{Lk + Lk)ExkVRk{Lk + Lk)Exk\ + E {x^Gnxn) + {ExnYGn^xn 



N-l 



= ^E[xl {Qk + LlRkLk) Xk + [Exkf^kExk] + E (xJ^Gnxn) + {ExNfCN'^XN (^.4) 



where 



A:=0 
N-l 

{^"^ [(Qfc + LlRkLk) E (xkxl)] + Tr {Exk{Exkf)] } 

A:=0 

+Tr [GnE (xnxJj)] + Tr [Gn {ExN{ExNf)] , 



ifc = Qk + [Lk + LkfRkiLk + Lk) + LlRkLk + IJi?fc£fc + Z^-RfcZfc. 



From the form (3.2) of the control, we may view {{Lk, Lk), k G N} as the new control input. Also 
(3.4) reminds us that Exk{Exk)'^ , E^Xkxl) may be considered as the new system states. Write Xk = 
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(3.5) 



E (xkxl) ,Xk = ExkiExkV. Then by (3.3), we have 

' Xk+i^ (Ak + BkLk)Xk [Ak + BkLkY + (^fe + BkLk)Xk [Ak + BkLk + Bk [L^ + Z^)] ^ 
+ [Ak + Bulk + Bk{Lk + Lk)] Xk{Ak + BkLkf 
+ [Ak + Bklk + Bk{Lk + Lk)] Xk [Ak + Bklk + Bk{Lk + Lk)f 
+ iCk + DkLk)Xk{Ck + DkLkV + {Ck + DkLk)Xk [Ck + DkU + Dk{Li + Lk)]^ 
+ [Ck + DkLk + Dk{Lk + Lk)] XkiCk + DkLk)'^ 
+ [Ck + DkLk + Dk{Lk + Lk)] Xk [Ck + DkLk + Dk(Lk + Lk)] 
= Xk{Lk, Lk), 
Xo= E(CC^), 
and 

Xk+i = [{Ak + Ak) + (Bk + Bk){Lk + Lk)] Xk [{Ak + Ak) + {Bk + Bk){Lk + Lk)]^ 

= Xk{Lk,Lk), (3.6) 
Xo = EC (EC)"^ . 

In the language of X and X, J(C, u) with it defined in (3.2) may be represented as 

N-l 

J(C, n) = i^'' [(Qfe + LlRkLk) Xk] + Tr } + Tr {GnXn) + Tr (GjvXfe) 

fe=0 

= J{Xo,Xo,C), (3.7) 
where C = {Lk,Lk,k G N}. Therefore, Problem (MF-LQ) is equivalent to the following problem: 



min J{Xo,Xo,£) 
subject to (3.5)(3.6). 



(3.8) 



Clearly, this is a matrix dynamic optimization problem. A natural way to deal with this class of problems 
is by the matrix minimum principle ([5]). Following the framework above, we have the following results. 

Theorem 3.1 For Problem (MF-LQ), under the condition 

Qk,Qk + Qk>Q, Rk,Rk + Rk>Q, k eN, Gn,Gn + Gn>Q, (3.9) 
the unique optimal control is 



K = -{w'^Y'H^^xk + [-{wn-'Hi," + {wn-'Hi,' 

= Llxk + Ll¥.xk, fcGN. 

Here, 

M^f ^ = + BlPk+iBk + DlPk+iDk, 

VFf ^ ^Rk+Rk + {Bk + Bk)^{Pk+i + Pk+i){Bk + Bk) + {Dk + Dk)^Pk+i{Dk + Dk), 
h\}^ = BlPk+iAk + DlPk+iCk, 
y iff) = {Bk + Bfe)^(Pfe+i + Pk+i){Ak + Ak) + {Dk + Dk)^Pk+i{Ck + Ck), 
with 



,(2)s i„(2) ^ 1^(1) 



(3.10) 



(3.11) 



Pk^Qk + LfRkLl + {Ak + BkLl)TPk+i{Ak + BkL^) + {Ck + DkLl)'^ Pk+i{Ck + DkLl), 
Pn = Gjv, 
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Pk 



Qk + Lh^PkLl + LfRkLl 

rO\Tl 



-{Ak + BkLiy Pu+i[Ak + Bull + Bk{Ll + L°)] 
+ Bull + Bk{Ll + Ll)]^Pk+i{Ak + BkLl) 
-[Au + Bull + Bu{Ll + ll)VPk+i[Au + Bull + MLl + H)] 
-{Ck + DuLl)^Pk+i[Cu + Dull + Du{Ll + ll)] 
iCk + Dull + Du{Ll + ll)fPu+i{Cu + DuLl) 
-[Cu + Dull + Du{Ll + ll)]'^Pu+i[Cu + Dull + Wl + ll)] 



, Pn = 
having the property 



+ [Au +Au + {Bu + Bu){Ll + L°)]^A+i[Afc + Au 
Gn, 



Pk,Pk + Pk>0, keN. 



(Bu+Bu){Ll + ll)], 



Proof. Introduce the Lagrangian function associated with Problem (3.8), 



71-1 



£ = ^ £fe + Tr (GnXn) + Tr {GnXu 



k=0 
n-1 



Tr 



fc=o 



(Gn G 



N 



Xn 
Xn 



where 

&k = Tr [{Qu + LlRuLu) Xu] + Tr {^uXu) 

+Tr [Pu+i {Xu{Lu,lu)-Xu+i)] +Tr [Pu+i {XuiLu, lu) ~ Xu+i)] 



Tr [{Qu + LlRuLu) Xu] + Tr {^uXu) + Tr 



'k+l Pk+1 



Xu{Lu, Lu) — Xu+1 
Xu{Lu, Lu) — Xu+i 



and {Pu+i Pk+i)ik G N are the Lagrangian multiphers. Denote 



1 = (Pu+i P, 



k+l 



Clearly, by the matrix minimum principle ([5]), the optimal feedback gains {Ll, Ll), k € N and Lag 
multipliers Pfe+i, fc 6 N satisfy the following first-order necessary conditions 

Pw = [Gn Gn) , 



dLu ' dLu _ ' dXu 
Pn = Gn, Pn = Gat. 



Pu 



d£u 
dXu 



fc eN, 
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Now, we should calculate several gradient matrices. Noting that for any matrix Y, ■^Tr{AYB) = A^B^ if 
AY B is meaningful, we have 



= {2RkLkXk + 2 [Rklk + Rk{Lk + Lk)] Xu] 

+2 {(Bfc + BkfPk+i{Ak + Ak) + {Bk + BkfPk+i{Bk + Bk){Lk + U)] 
+2 {BlPk+i{Ak + BkLk)Xk + BlPk+i{Ak + BkLk)Xk + B^Pk+iiAk + BkLk)Xk 
+BlPk+iBk{Lk + Lk)Xk + BlPk+i{Ak + BkLk)Xk + B^Pk+iBkiLk + Lk)Xk} 
+2 {DlPk+i{Ck + DkLk)Xk + DlPk+i{Ck + DkLk)Xk + DlPu+i{Ck + DkLk)Xk 
+DlPk+iDk{Lk + Lk)Xk + DlPk+i{Ck + DkLk)Xk + DlPk+MLk + 

= 2 [RkLk + BlPk+i{Ak + BkLk) + DlPk+i{Ck + DkLk)] Xk 



d^,Vp, 

T 



k+1 



-2 { [Rk + Rk + (Bk + BkfiPk+i + Pk+i){Bk + Bk) + {Dk 
- [Rk + (Bk + BkfiPk+i + Pk+i)iBk + Bu) - BlPk+iBk + DlPk+iD 
^DlPk+iDk + DlPk+iDk] + [Bk + Bk f{Pk+i + Pk+i){Ak + Ak) 
V{Dk + DkfPk+iiCk + Ck) - BlPk+iAk - DlPk+iCk] Xk 



{Dk + Dk)] Lk 



2 ( ^Lfc 



2 { wl^'>Lk + H)^^ ){Xk-Xk) + 2[ Wj:'' {Lk + Lk) + Hi:'> ) Xk 



Hi}^] Xk 
r(i) 



2 ( ^Lfe 



WSL 



(3) 



Xk 



.(2) 



r(2) 



(3.18) 



- 2{[Rk + Rk + {Bk + Bk)^{Pk+i + Pk+i){Bk +B-k) + {Dk + Dk)^Pk+i{Dk + Dk)] {Lk + Lk 

+{Bk + Bk)^{Pk+i + Pk+i){Ak + Ak) + {Dk + Dk)'^Pk+i{Ck + Ck)}Xk 
= 2(w^f (i, + L,)+<)x,. 

Here wl^\H\^\i,j = 1,2, are defined in (3.11), and 



(3.19) 



(3) 



(3) 



Rk + {Bk + Bk)^{Pk+i + Pk+i){Bk + Bk) - BlPk+iBk + D^Pk+iDk 
+DlPk+iDk + DlPk+iDk, 

{Bk + Bk)'^{Pk+i + Pk+i){Ak + Ak) + {Dk + Dk)^Pk+i{Ck + Ck) 
-BlPk+iAk-DlPk+iCk. 



The following properties are used 



(2) 



.(1) 



.(3) 
'k ' 



(2) 



(1) 



(3) 



Combining (3.17)-(3.19), the optimal feedback gains and L% must satisfy 
{wi^'^Lk + iff ^) {Xk ~ Xk) + (w/f ^ {Lk + Lk 



-(2) fr. ^f.\^ H-(2) 



w^^> {Lk + Lk) + nr ]Xk^O. 



k) 

Xk 



) Xk 



0, 



(3.20) 



Note that (3.20) holds for any initial values Xq-Xq = E [(( - EC)(C - EC)^] and Xq = EC (EC)^. Therefore, 
(3.20) reduces to 



wi''Lk + H\^^ 



0, 



(3.21) 
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which is obtained by letting coefficients be zero in (3.20). Clearly, we obtain the optimal feedback gains 



LI 
LI 



r(2) 



r(i) 



We now derive the equations that and Pk satisfy. By (3.17), we have 



Pk = 



Pk 



L'k—LZ^Lu—LZ 



dXk 



-{A, + BkLlfPk+i [Ak + Bill + Bk{Ll + Z^)] 
+ [Au + BkLl + Bk{Ll + Ll)fPu+i{Ak + BkLl) 
+ [A, + Bkll + Bk{Ll + Ll)fPk+i [Ak + Bk'Ll + B,(L°. + Z°)] 
+ {Ck + DkLl)^Pk+i [Ck + Dkll + Dk{Ll + LI)] 
+ [Ck + Dkll + Dk{Ll + Ll)fPk+i{Ck + DkLl) 
+ [Cfe + Dkll + ^fe(L° + ll)YPk+i [Ck + + L)k{Ll + ID] 

+ [(Afe + Ak) + (Sfe + Bk){Ll + Z°)]^ A+i [(Afc + Ak) + (Bfc + Bfe)(i2 + ID] , 

which are (3.12) and (3.13). The final thing is to assure that for any A: G N, Pk, Pk + Pk > 0. We prove this 
by induction. Clearly, P/v, Pn + Pn > by definition. For k ^ N — 1, Pn-i is positive semi-definite, while 

Pn-1 + Pn-1 = Qn-1 + Qn-1 + {L%_i + l%_i)'^ Rn-i{L%_i + l^^i) 

+ [A + A+iB + B)iL%_, + l%_,)f{PM + Pn)[A + A+{B + B){L%_, + l%_,)] 

+ [C + C+{D + D){L%_, + l%_^)fPN[C + C+iD + D){L%_, + l%_^)] > 0. 

In addition, by (3.10), we have 

jjy_l(.T7V-l,lt°) 

= E {x%_^Qn-ixn-i) + (Ea;Ar_i)^(5jv-iEa:Ar_i + E {u]^_iRn-iUn-i) 

+{Kun^i)^ Rpf^iEuN^i + E (^xJjGnxn) + {^xn)'^GnKxn 
= E [x%_^ {Qn-1 + L°^_^Rn-iL%_^) x] 

+(Ex)^ [Qat-i + L°^_-^Rm-iI%^i + 1°^_'^^Rn-iL%_i + 1'^_iRn-iI'n~i 

+ {L%f_i + 1% _i)'^ Rn -i{L% _i + l%^i)] Ea;Ar_i 



-E 



({A + BL''^_Y)^Pn{A + BL^_i) + (C + DL%_J''Pn{C + DL%_,))xn-i 
+ iKxN-i)^ [{A + BL%_,)^Pn[A + Bl%_, + B{L%_, + l%_,)] 
+ [A + Bl%_^ + B(£^_i + 1%_^)YPn{A + BL%_Y) 
+ [A + Bl%_^ + B{L%_^ + 1°^_Y)VPn[A + Bll 



-iC- 
-[C- 



Dl^N-l 

Dl%^, 



Pn[C + DL%_^ + D{L%_^ + L^_i)] 
D{L'},_, + 1%_Y,VPn{C + DL%_,) 
D{L%_, + 11_Y,VPn[C + Dll_^ + D{L%_, + ll_Y)] 



+ [A + A+{B + B){L%_^ + 1%_^)\^Pn[A + A+{B + B){L%_, + l"j^_,)] 

E {x'j^_^PN^iXN^i) + (Ex7V-l)^ Pn-1 (El-AT-i) . 



Exn^ 
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Noting that 

Jf (xfc,?/°|{i,fe+i,..,jv-i}) = E [{xlQkXi + {Ex^fQkExk + K)TRkul + (Eulf RkEul)] 

+ Jk+l{^k+l,U°\[k+l,k+2,--- ,N-1}), 

by induction, we have that Pk, Pk + Pk > 0, for any fc G N. This completes the proof. 



□ 



We may view Problem (MF-LQ) another way. For any z ^ Ti = Xk,l^k, easily see that Ez is orthogonal 
to z — Ez, as 

{Ez, z - Ez) = E {Ezf (z - Ez) = 0. 

Thus, (3.2) is equivalent to 

Uk = (Lfe + Lk)Exk + Lk{xk - Exk) = MkExk + Lk{xk -Exk), (3.22) 

which is a function of Exk and Xk — Exk- Clearly, in (3.22), we may design Mk and Lk independently. This 
reminds us that we may study Problem (MF-LQ) using coordinates {Exk,Xk — Exk} for any fc G N. The 
system equations that Exfc, Xk — Exk satisfy are 



(3.23) 



Exk+i = {Ak + Ak)Exk + {Bk + Bk)Euk, 
Exo = EC, 

Xk+i - Exk+i ^ [Ak{xk - Exk) + Bk{uk - Euk)] + [Ck{,Xk - Exk) + {Ck + Ck)Exk 

+Dk{uk - Euk) + {Dk + Dk)Euk]wk. (3.24) 
xa - E.To = C - EC. 

The cost functional J{(^,u) may be represented as 

N-l 

J(C, u)= ^ E [(.Tfc - Exkf Qk {xk - Exk) + {Exkf {Qk + Qk)Exk + {uk - Eukf Rk {uk - Eufe) 



fe=0 



(3.25) 



[EukY {Rk + Rk)^uk 



{xn — Exn)^ Gn {xn — Ea^Af) + (E.tjv)'^ (Gat + GAr)Ea;Ar 
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To follow the classical method of completing the square, we introduce two sequences of symmetric matrices 
{Sk, fc G N} and {Tk, fc G N}, which are determined below. Then, 

J(C,m) + (.Tjv — E^at) Sn {xn — E.TAr) — (.To — Exq) 5*0 {xq — E.To) + (Exjv) TatExjv — (Exq) TqExo 

N-1 

= ^ E [(xfc - Exkf Qk (xk - Exfc) + {Exkf [Qk + Qk)Exk + [uk - Eukf Rk {uk - Euk) 

+ (Eufe) {Rk + RkW-Uk + ixk+i-Exk+i) S'fe+i (xfe+i - Eccfc+i) - (xfe - Exfe) Sk{xk~Exk) 



{Exk+iY Tk+iExk+i - {ExkY Tk&Xk 



+ (Ea;Ar)^ {Gn + Gn)Exn 

N-1 



(xat— Exjv) Gat (xjv — E.Tjv) 



E [(xfc - E.Tfe)^ {Qk + AlSk+iAk + ClSk+iCk) {xk - Exk) 

k=0 

+2 [xk - Exkf {AlSk+iBk + ClSk+iDk - Sk) [uk - Euk) 
+ {uk - Eukf {Rk + BlSk+iBk + DlSk+iDk) {uk - Euk) 

+ {Exkf {Qk + Qk + {Ck + Ck)^Sk+iiGk + Ck) + (Ak + AkfTk+i{Ak + Ak) - Tk) Exk 
+2 {Exk)'^ {{Ck + CkVSk+i{Dk + Dk) + {Ak + Ak)Tk+i{Bk + Bk)) Euk 
+ (Eufe)^ {Rk + Rk + {Dk + Dk)^Sk+i{Dk + Dk) + {Bk + Bk)^Tk+i{Bk + Bk)) Euk 

+E ^{xn — Exn)'^ Gn {xn — Exn) + (Exm)"^ {Gn + Gn)Exn 



N-1 



E.Tfe) 



fc=0 



AlSk+iAk + GlSk+iCk ~Sk~ {Hl'Y{wt'Y'Hl'^ 



{xk - Exk) 



Uk - Euk + {wlY^)-^H'k' {xk - Esfe) j Wl^'> [uk - Euk + {Wl^'')-^Hl'' {xk - Exk) 
+ {Exk)'^ {Qk + Qk + {Gk + Ck)'^Sk+i{Gk + Ck) + {Ak + Ak)'^Tk+i{Ak + Ak) - Tk) Exk 



(1) 



^(l)^-lH-(l) 



-(Exk)^{HPV(Wn-'Hi: 



r(2) 



iXk 



+ (Euk + {wlY^)-'Hi^'>Exk) W-f ^ (Euk + (W-i'V^fl-f ^Exfc 



-E 



{xn —Exn) Gn {xn — ^xn) + (Exn) {Gn + Gn)^xn 



Here 



— Rk + B^Sk+iBk + DJ^ Sk+iDk, 



h'"^^ = Bj. Sk+iAk + Dj, Sk+iCk, 

) =Rk + Rk + {Bk + Bk)^Tk+i{Bk + Bk) + {Dk + DkfSk+i{Dk + Dk), 



(2) 



{Bk + Bk)^Tk+i{Ak + Ak) + {Dk + DkYSk+i{Ck + Ck). 



(3.26) 



Letting 



Sk - 
Tk -- 
Sn 



Qk + AlSk+iAk + ClSk+iCk - {h\^^)^{wIY^)-'h]'^ 



k ' 



Qk + Qfc + {Ck + Ck)^Sk+i{Ck + Ck) + {Ak + AkfTk+Mk + Ak) - (i/f )^(VKf \(3-27) 



G 



N: 



T 



N 



Gpf + G 



N- 



and 



Uk - Euk = -(W^i'V'i?!'^ {xk - Kxk) , fc G N, 



Euk 



(V^fV^fExfe, fcGN, 



that is, 



Uk 



-(W^f V'^f ^Exfe - (W^i'V'ffi'^ {xk - Exk) , fc G N, 



(3.28) 
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we have 



J(C,S) = E (C-EC)^5o(C-EC) + (EC)^ToEC < J{C,u) 



for any u = {ug, ■ ■ ■ ,un-i) with Uk G Uk- This means that (3.28) is an optimal control. In the following, 
wc show that (3.28) is equal to (3.10). We need firstly to show that for any fc G N, = Sk,Tk = Pk + Pk- 
In fact, by substituting (3.10) into (3.12), we have 

Pk = Qk + AlPk+iAk + ClPk+iCk + {Llf {Rk + BlPk+iBk + D^Pk+MLl 



{LlfHi'^ + (Hi'^) LI (3.29) 



T 



Qk + AlPk+,Ak + ClPk+rCk - {Hi'^Viwt'Y'Hi'^ 



Noting that Pjv = Ga? = S'jv, by (3.11)(3.26)(3.27)(3.29), we have that 

Sk^Pk, Hi^^=Hi^\ k€N. 

Similarly, we have by (3. 13) (3. 12) 

Pk = Qk - LfRkLl - {Ak + BkLlf Sk+i {Ak + BkLl) - {Ck + DkLlf Sk+i {Ck + DkL%) 

+{Ck + CkVSk+i{Ck + Ck) + {Ak + AkfiPk+i + Pk+i){Ak + Ak) - (i?f )^(W^f )"'4'^ 
= Qk + Qk- Pk + [Ck + Ck)'^Sk+i{Ck + Ck) + {Ak + Ak)^{Pk+i + Pk+i){Ak + Ak) 

Therefore, 

Pk + Pk = Qk + Qk + [Ck + Ck)'^Sk+iiCk + Ck) + (Ak + Ak)^{Pk+i + Pk+i)iAk + Ak) 

-(iff )^(I^f' )-'^r- (3.30) 
Comparing this with (3.27), as T/v = Gn + Gn = Pn + Pn, we have that 

Tk^Pk + Pk, VF^ ^ = M^f \ = hI^\ keN. 

Therefore, (3.28) equals to (3.10). 

To summarize. Theorem 3.1 may be rewritten in a more compact form as the following results: 

Theorem 3.2 For Problem (MF-LQ), under the condition 

Qk,Qk + Qk>0, Rk,Rk + Rk>0, keN, Gn,Gn + Gn>0, (3.31) 

the optimal control is 

< = -(W^f V^^Exfe - (I^i' V^ff ^ (^fc - Exfc) = M^Exk + Ll{xk - Exfc). (3.32) 

Here, 

W^^^ = Rk + BlSk+iBk + DlSk+iDk, 

lyf ^ = Rk + Rk + {Bk + Bk)^Tk+i{Bk + Bk) + {Dk + Dk)^ Sk+i{Dk + Dk), 
H^k^ ~ Bj:Sk+iAk + D^Sk+iCk, 

ijf ) = {Bk + Bk)^Tk+Mk + Ak) + {Dk + Dk)^Sk+i{Ck + Ck), (3-33) 
Sk = Qk + AlSk+iAk + ClSk+iCk - {Hi'^)^{wj,'^)-^Hi'\ 

Tk = Qk + Qk + {Ck + CkfSk+i{Ck + Ck) + {Ak + Ak)^Tk+i{Ak + Ak) - (i/f ^ (V^f V^i/f \ 
Spf ~ Gn, T/v = Gn + Gn- 

with the following property 

Pk,Tk>0, fcGN. (3.34) 
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4 Numerical results 



Wc consider a 4-pcriod numerical example 

3 



mm 

Uo,Ui,U2,U3 



E 



.k=0 



+ (Ex4)^G4Ea:;4, 



, , Xk+i = {AkXk + AkExk + BkUk + BkEuk) + {CkXk + CkExk + DkUu + DkEuk)wk, 

subject to <; ^^^3 A; = 0,1, 2, 3, 



xo e 



with coefficients for /c = 0, 1, 2, 3 as follows 





' 0.2 


0.4 


0.2 " 




" 0.3 


0.4 


0.2 


Ak = 





0.2 


0.6 


Ak = 





0.2 


0.7 




_ 0.6 


0.4 


0.2 _ 




_ 0.6 


0.5 


0.2 




" 0.4 


0.2 " 






" 0.5 


0.2 " 




Bk = 


0.2 


0.4 




Bk - 


0.2 


0.5 






0.3 


0.3 






0.2 


0.3 







' 0.2 


0.4 


0.6 " 




" 0.3 


0.4 


0.6 


Ck = 


0.4 


0.2 


0.6 


, Ck = 


0.4 


0.3 


0.6 




0.2 


0.4 


0.2 




0.2 


0.4 


0.3 




" 0.2 


0.6 " 






" 0.3 


0.5 " 




Dk = 


0.6 


0.4 




Dk - 


0.5 


0.4 






0.3 


0.1 






0.3 


0.3 





Qk 

Rk 
G4 



= diag([0, 1.5,1] 
= diag([l,l]), 
= diag([0,l,l]). 



Qfc = diag([l,l,0]), 
=diag([1.5,l]), 
G4 = diag([0.5,l,0]). 



Based on the Riccati equations (3.33), we have Riceati solutions for Si and Ti for i = 0, 1, 2, 3 are given by 





" 0.5227 





3542 





1966 " 






4.3329 


1.7927 


-0.2507 


So = 


0.3542 


1 


9655 





3170 






1.7927 


4.4213 


0.4463 




0.1966 





3170 


1 


7009 






0.2507 


0.4463 


3.4720 




" 0.5188 





3513 





1951 ■ 






4.2341 


1.7868 


-0.2366 


Si = 


0.3513 


1 


9595 





3130 






1.7868 


4.4007 


0.4611 




0.1951 





3130 


1 


6943 






0.2366 


0.4611 


3.4411 




" 0.4862 





3264 





1861 " 






3.4908 


1.6119 


-0.0389 


S2 = 


0.3264 


1 


9219 





2928 


T2 = 




1.6119 


4.2394 


0.4881 




0.1861 





2928 


1 


6660 






0.0389 


0.4881 


3.3283 




" 0.3747 





2421 





1492 " 




" 1 


4782 


0.3777 


0.3734 " 






0.2421 


1 


7652 





1849 







3777 


3.2001 


0.5548 






0.1492 





1849 


1 


4532 







3734 


0.5548 


2.4932 





Then applying Theorem 3.2, we get the optimal control below 



^M^Exk+Ll{xk-Exk), fc = 0,l,2,3, 
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where 



Mo = 


■ -0.3286 
-0.3189 


-0.4234 
-0.4351 


-0.3474 ■ 
-0.7770 


T " — 


■ -0.3455 
-0.2467 


-0.3271 
-0.2937 


-0.4240 " 
-0.4941 


Ml = 


■ -0.3436 
-0.3137 


-0.4156 
-0.4381 


-0.3531 ■ 
-0.7687 




■ -0.3436 
-0.2446 


-0.3235 
-0.2897 


-0.4207 " 
-0.4885 


M| = 


■ -0.4029 
-0.2938 


-0.3946 
-0.4160 


-0.3315 ■ 
-0.7519 




■ -0.3290 
-0.2298 


-0.3009 
-0.2692 


-0.4043 " 
-0.4650 


Af 3° = 


■ -0.2418 


-0.2552 


-0.3178 ■ 




■ -0.2552 


-0.2084 


-0.2954 " 


-0.1351 


-0.2213 


-0.5101 


-0.1744 


-0.1608 


-0.3028 



5 Conclusion 

In this paper, we give four methods to deal with the discrete time mean-field LQ problem: the quadratic 
optimization method in Hilbert space, the operator LQ method, the matrix dynamic optimization method 
and the method of completing the square. The optimal control is a linear state feedback using two Riccati 
equation. For future research, we may consider an infinite horizon mean-field LQ problem. In that case, the 
stability of the system should first be considered. We shall investigate this in the near future. 
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